Papers with pruning method
Task-specific Compression for Multi-task Language Models using Attribution-based Pruning (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing compression methods for multi-task language models use large number of parameter parameters even when performing only a specific task. |
| Approach: | They propose a training-free compression method for multi-task language models using pruning method . they use an attribution method to determine which neurons are essential for performing a specific task . |
| Outcome: | The proposed method outperforms baseline pruning methods on six widely-used datasets. |
Two-Stage Regularization-Based Structured Pruning for LLMs (2026.acl-long)
Copied to clipboard
Mingkuan Feng, Jinyang Wu, Siyuan Liu, Shuai Zhang, Hongjian Fang, Ruihan Jin, Feihu Che, Pengpeng Shao, Zhengqi Wen, Jianhua Tao
| Challenge: | Structural pruning is a promising solution for large language models . prior structured pruning methods remove unimportant parameters based on certain metrics . |
| Approach: | They propose a structural pruning method that iteratively learns the weights of transformer layers by adding their l1-norm to the loss function. |
| Outcome: | The proposed pruning method outperforms strong layer-wise pruning methods without requiring retraining. |
Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Model (2023.acl-long)
Copied to clipboard
| Challenge: | Neural Machine Translation models are based on a Mixture of Experts architecture and can be pruned to remove up to 80% of experts without further finetuning. |
| Approach: | They propose a pruning method that removes up to 80% of experts without further finetuning and with a negligible loss in translation quality. |
| Outcome: | The proposed pruning method removes up to 80% of experts without further finetuning and with a negligible loss in translation quality. |
The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Models (2022.emnlp-main)
Copied to clipboard
Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, Dan Alistarh
| Challenge: | Pre-trained Transformer models provide robust language representations which can be specialized on various tasks. |
| Approach: | They propose an efficient pruning method based on approximate second-order information that allows pruning weight blocks to be used for pruning. |
| Outcome: | The proposed method is the first to be applied at the BERT scale and significantly pushes the boundaries of the current sparse models with respect to all metrics: model size, inference speed and task accuracy. |
RASD: Retrieval-Augmented Speculative Decoding (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for generating draft tokens rely on lightweight draft models or additional model structures to generate tokens and retrieve context from databases. |
| Approach: | They propose to use a pruning method to enhance model-based speculative decoding by combining the best-fit model with the best retrieval tree. |
| Outcome: | The proposed method achieves state-of-the-art inference acceleration across tasks such as DocQA, Summary, Code, and In-Domain QA. |
One Network, Many Masks: Towards More Parameter-Efficient Transfer Learning (2023.acl-long)
Copied to clipboard
| Challenge: | Parameter-efficient transfer learning methods can be expensive in storage when applied to broader ranges of tasks. |
| Approach: | They propose a method that enables efficient sharing of a single PETL network across layers and tasks. |
| Outcome: | The proposed method outperforms other methods with 10% parameters required by the latter on various downstream tasks. |
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned (P19-1)
Copied to clipboard
| Challenge: | et al., 2017) show that multi-head attention is important for neural machine translation. |
| Approach: | They evaluate the contribution made by individual attention heads to the overall performance of the Transformer model and analyze the roles played by them in the encoder. |
| Outcome: | The proposed pruning method removes the vast majority of heads without affecting performance. |
LLM-TOPLA: Efficient LLM Ensemble by Maximising Diversity (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Extensive evaluation of modern large language models shows performance gain over component LLMs. |
| Approach: | They propose a diversityoptimized LLM ensemble method with three unique properties . they introduce the focal diversity metric to capture diversityperformance correlation . |
| Outcome: | The proposed method outperforms the best-performing ensemble on four benchmarks. |
When Compression Meets Model Compression: Memory-Efficient Double Compression for Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) exhibit excellent performance in various tasks, but memory requirements present a challenge when deploying on memory-limited devices. |
| Approach: | They propose a framework to compress LLM after quantization further, achieving about 2.2x compression ratio. |
| Outcome: | The proposed model can achieve 40% reduction in memory size with negligible loss in accuracy and inference speed. |
M-Wanda: Improving One-Shot Pruning for Multilingual LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual LLM performance is often dependent on model size, resulting in performance loss. |
| Approach: | They propose a pruning method that models cross-lingual variation by incorporating language-aware activation statistics into its pruning criterion and dynamically adjusts layerwise sparsity based on cross-linguistic importance. |
| Outcome: | The proposed method improves performance at minimal additional costs while maintaining multilinguality. |
PDTrim: Targeted Pruning for Prefill-Decode Disaggregation in Inference (2026.acl-long)
Copied to clipboard
| Challenge: | Existing pruning methods ignore prefill-decode (PD) disaggregation in practice. |
| Approach: | They propose a pruning method that is highly integrated with prefill-decode (PD) disaggregation, enabling more precise pruning of blocks. |
| Outcome: | The proposed method achieves strong performance in both PD disaggregation and PD unified settings, and can be extended to other non-block pruning methods. |